Providing features of an electronic product in an augmented reality environment

ABSTRACT

The present disclosure relates to providing a software feature of an electronic product in an augmented reality (AR) environment. In some embodiments, images are obtained using one or more image sensors, a determination is made whether the obtained images include printed media depicting the electronic product, when the obtained images include the printed media depicting the electronic product, a virtual object corresponding to the electronic product is displayed in the AR environment, and the software feature of the electronic product is provided with the virtual object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/897,551, entitled “PROVIDING FEATURES OF AN ELECTRONIC PRODUCT IN ANAUGMENTED REALITY ENVIRONMENT,” filed on Jun. 10, 2020, which is acontinuation of U.S. patent application Ser. No. 16/140,286, entitled“PROVIDING FEATURES OF AN ELECTRONIC PRODUCT IN AN AUGMENTED REALITYENVIRONMENT,” filed on Sep. 24, 2018, which claims the benefit of U.S.Provisional Patent Application No. 62/565,668, entitled “PROVIDINGFEATURES OF AN ELECTRONIC PRODUCT IN AN AUGMENTED REALITY ENVIRONMENT,”filed on Sep. 29, 2017, which are hereby incorporated by reference intheir entirety.

TECHNICAL FIELD

The present disclosure relates generally to augmented realityenvironments, and more specifically to representations of electronicproducts in augmented reality environments.

BACKGROUND

Printed media are physical objects that include depictions of textand/or images. Examples of printed media include books, magazines,newspapers, flyers, billboards, posters, and the like. In some cases,printed media include depictions of physical, real-world products, suchas mobile phones, computers, televisions, and so on.

SUMMARY

Described herein are techniques for providing features of an electronicproduct in an augmented reality (AR) environment. In some embodiments, atechnique for providing software features of the electronic product inthe AR environment includes obtaining images using one or more imagesensors; determining whether the obtained images include printed mediadepicting the electronic product; in accordance with a first set of oneor more conditions being satisfied, the first set of one or moreconditions including a first condition that is satisfied when theobtained images include the printed media depicting the electronicproduct, displaying a virtual object corresponding to the electronicproduct in the AR environment; and in accordance with a second set ofone or more conditions being satisfied, the second set of one or moreconditions including a second condition that is satisfied when thevirtual object corresponding to the electronic product is displayed,providing, with the virtual object, the software feature of theelectronic product.

In some embodiments, the technique further includes detecting a userinteraction with the software feature; and modifying the virtual objectcorresponding to the electronic product based on the user interaction.In some embodiments, detecting the user interaction includes detecting atouch contact on a touch-screen display at a location corresponding tothe provided software feature. In some embodiments, detecting the userinteraction includes detecting that a physical position of the user'shand or finger corresponds to a position in the AR environment of thesoftware feature.

In some embodiments, the first set of one or more conditions include athird condition that is satisfied when the printed media corresponds topredefined printed media. In some embodiments, the predefined printedmedia is a predefined book, a predefined page of a predefined book, or apredefined depiction of the electronic product. In some embodiments, thefirst set of one or more conditions include a fifth condition that issatisfied when the depiction of the electronic product in the printedmedia includes a depiction of a screen of the electronic product. Insome embodiments, the first set of one or more conditions include asixth condition that is satisfied when a user input is detected.

In some embodiments, the second set of one or more conditions include aseventh condition that is satisfied when a user interaction with thevirtual object is detected.

In some embodiments, providing the software feature includes displaying,with the virtual object corresponding to the electronic product, anartificial boot-up sequence that depicts a boot-up sequence of theelectronic product. In some embodiments, providing the software featureincludes displaying, with the virtual object corresponding to theelectronic product, a video depicting one or more aspects of thesoftware feature.

In some embodiments, a device for providing the software feature of theelectronic product in the AR environment includes one or more processorsand memory storing one or more programs configured to be executed by theone or more processors. The one or more programs include instructionsfor obtaining images using one or more image sensors; determiningwhether the obtained images include printed media depicting theelectronic product; in accordance with a first set of one or moreconditions being satisfied, the first set of one or more conditionsincluding a first condition that is satisfied when the obtained imagesinclude the printed media depicting the electronic product, displaying avirtual object corresponding to the electronic product in the ARenvironment; and in accordance with a second set of one or moreconditions being satisfied, the second set of one or more conditionsincluding a second condition that is satisfied when the virtual objectcorresponding to the electronic product is displayed, providing, withthe virtual object, the software feature of the electronic product.

In some embodiments, a non-transitory (or, optionally, transitory)computer-readable storage medium storing one or more programs configuredto be executed by one or more processors provides the software featureof the electronic product in the AR environment. The one or moreprograms include instructions for obtaining images using one or moreimage sensors; determining whether the obtained images include printedmedia depicting the electronic product; in accordance with a first setof one or more conditions being satisfied, the first set of one or moreconditions including a first condition that is satisfied when theobtained images include the printed media depicting the electronicproduct, displaying a virtual object corresponding to the electronicproduct in the AR environment; and in accordance with a second set ofone or more conditions being satisfied, the second set of one or moreconditions including a second condition that is satisfied when thevirtual object corresponding to the electronic product is displayed,providing, with the virtual object, the software feature of theelectronic product.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following description, reference is made to the accompanyingdrawings which form a part thereof, and which illustrate severalembodiments of the present disclosure. It is understood that otherembodiments may be utilized and structural and operational changes maybe made without departing from the scope of the present disclosure. Theuse of the same reference symbols in different drawings indicatessimilar or identical items.

FIGS. 1A-1B depict exemplary systems for use in variouscomputer-generated reality technologies, including virtual reality andmixed reality.

FIGS. 1C-1E illustrate embodiments of the system in the form of mobiledevices.

FIGS. 1F-1H illustrate embodiments of the system in the form ofhead-mounted display devices.

FIG. 1I illustrates an embodiment of the system in the form of a head-updisplay device.

FIG. 2 illustrates an example of physical printed media.

FIG. 3 illustrates an embodiment of a device displaying a representationof printed media.

FIG. 4 illustrates an embodiment of a device displaying a virtualelectronic product overlaying a representation of printed media.

FIG. 5 illustrates an embodiment of a device displaying a virtualelectronic product after a user interaction with an emulated softwarefeature.

FIG. 6 illustrates another embodiment of a device displaying a virtualelectronic product overlaying a representation of printed media.

FIG. 7 illustrates an exemplary technique for providing a softwarefeature of an electronic product in an augmented reality environment.

DETAILED DESCRIPTION

Various embodiments of electronic systems and techniques for using suchsystems in relation to various computer-generated reality technologies,including virtual reality and mixed reality (which incorporates sensoryinputs from a physical environment), are described. In particular, thepresent disclosure provides techniques for displaying a virtual objectcorresponding to an electronic product in an AR environment. Theelectronic product is identified in physical printed media (e.g.,magazine, newspaper, flyer, billboard, poster, and so forth), and then avirtual object corresponding to the electronic product is displayed inthe AR environment. The virtual object includes one or more softwarefeatures (e.g., boot-up sequence, user interfaces) of the physicalelectronic product. User interactions with the virtual object cause thevirtual object to be modified in a manner resembling the physicalelectronic product. For example, a user may interact with a userinterface displayed on virtual mobile phone in a similar manner as wouldbe done on a real-world user interface of a physical mobile phone.

A physical environment (or real environment) refers to a physical worldthat people can sense and/or interact with without aid of electronicsystems. Physical environments, such as a physical park, includephysical articles (or physical objects or real objects), such asphysical trees, physical buildings, and physical people. People candirectly sense and/or interact with the physical environment, such asthrough sight, touch, hearing, taste, and smell.

In contrast, a computer-generated reality (CGR) environment refers to awholly or partially simulated environment that people sense and/orinteract with via an electronic system. In CGR, a subset of a person'sphysical motions, or representations thereof, are tracked, and, inresponse, one or more characteristics of one or more virtual objectssimulated in the CGR environment are adjusted in a manner that comportswith at least one law of physics. For example, a CGR system may detect aperson's head turning and, in response, adjust graphical content and anacoustic field presented to the person in a manner similar to how suchviews and sounds would change in a physical environment. In somesituations (e.g., for accessibility reasons), adjustments tocharacteristic(s) of virtual object(s) in a CGR environment may be madein response to representations of physical motions (e.g., vocalcommands).

A person may sense and/or interact with a CGR object using any one oftheir senses, including sight, sound, touch, taste, and smell. Forexample, a person may sense and/or interact with audio objects thatcreate a 3D or spatial audio environment that provides the perception ofpoint audio sources in 3D space. In another example, audio objects mayenable audio transparency, which selectively incorporates ambient soundsfrom the physical environment with or without computer-generated audio.In some CGR environments, a person may sense and/or interact only withaudio objects.

Examples of CGR include virtual reality and mixed reality.

A virtual reality (VR) environment (or virtual environment) refers to asimulated environment that is designed to be based entirely oncomputer-generated sensory inputs for one or more senses. A VRenvironment comprises a plurality of virtual objects with which a personmay sense and/or interact. For example, computer-generated imagery oftrees, buildings, and avatars representing people are examples ofvirtual objects. A person may sense and/or interact with virtual objectsin the VR environment through a simulation of the person's presencewithin the computer-generated environment, and/or through a simulationof a subset of the person's physical movements within thecomputer-generated environment.

In contrast to a VR environment, which is designed to be based entirelyon computer-generated sensory inputs, a mixed reality (MR) environmentrefers to a simulated environment that is designed to incorporatesensory inputs from the physical environment, or a representationthereof, in addition to including computer-generated sensory inputs(e.g., virtual objects). On a virtuality continuum, a mixed realityenvironment is anywhere between, but not including, a wholly physicalenvironment at one end and virtual reality environment at the other end.

In some MR environments, computer-generated sensory inputs may respondto changes in sensory inputs from the physical environment. Also, someelectronic systems for presenting an MR environment may track locationand/or orientation with respect to the physical environment to enablevirtual objects to interact with real objects (that is, physicalarticles from the physical environment or representations thereof). Forexample, a system may account for movements so that a virtual treeappears stationary with respect to the physical ground.

Examples of mixed realities include augmented reality and augmentedvirtuality.

An augmented reality (AR) environment refers to a simulated environmentin which one or more virtual objects are superimposed over a physicalenvironment, or a representation thereof. For example, an electronicsystem for presenting an AR environment may have a transparent ortranslucent display through which a person may directly view thephysical environment. The system may be configured to present virtualobjects on the transparent or translucent display, so that a person,using the system, perceives the virtual objects superimposed over thephysical environment. Alternatively, a system may have an opaque displayand one or more imaging sensors that capture images or video of thephysical environment, which are representations of the physicalenvironment. The system composites the images or video with virtualobjects, and presents the composition on the opaque display. A person,using the system, indirectly views the physical environment by way ofthe images or video of the physical environment, and perceives thevirtual objects superimposed over the physical environment. As usedherein, a video of the physical environment shown on an opaque displayis called “pass-through video,” meaning a system uses one or more imagesensor(s) to capture images of the physical environment, and uses thoseimages in presenting the AR environment on the opaque display. Furtheralternatively, a system may have a projection system that projectsvirtual objects into the physical environment, for example, as ahologram or on a physical surface, so that a person, using the system,perceives the virtual objects superimposed over the physicalenvironment.

An augmented reality environment also refers to a simulated environmentin which a representation of a physical environment is transformed bycomputer-generated sensory information. For example, in providingpass-through video, a system may transform one or more sensor images toimpose a select perspective (e.g., viewpoint) different than theperspective captured by the imaging sensors. As another example, arepresentation of a physical environment may be transformed bygraphically modifying (e.g., enlarging) portions thereof, such that themodified portion may be representative but not photorealistic versionsof the originally captured images. As a further example, arepresentation of a physical environment may be transformed bygraphically eliminating or obfuscating portions thereof.

An augmented virtuality (AV) environment refers to a simulatedenvironment in which a virtual or computer generated environmentincorporates one or more sensory inputs from the physical environment.The sensory inputs may be representations of one or more characteristicsof the physical environment. For example, an AV park may have virtualtrees and virtual buildings, but people with faces photorealisticallyreproduced from images taken of physical people. As another example, avirtual object may adopt a shape or color of a physical article imagedby one or more imaging sensors. As a further example, a virtual objectmay adopt shadows consistent with the position of the sun in thephysical environment.

There are many different types of electronic systems that enable aperson to sense and/or interact with various CGR environments. Examplesinclude head mounted systems, projection-based systems, heads-updisplays (HUDs), vehicle windshields having integrated displaycapability, windows having integrated display capability, displaysformed as lenses designed to be placed on a person's eyes (e.g., similarto contact lenses), headphones/earphones, speaker arrays, input systems(e.g., wearable or handheld controllers with or without hapticfeedback), smartphones, tablets, and desktop/laptop computers. A headmounted system may have one or more speaker(s) and an integrated opaquedisplay. Alternatively, a head mounted system may be configured toaccept an external opaque display (e.g., a smartphone). The head mountedsystem may incorporate one or more imaging sensors to capture images orvideo of the physical environment, and/or one or more microphones tocapture audio of the physical environment. Rather than an opaquedisplay, a head mounted system may have a transparent or translucentdisplay. The transparent or translucent display may have a mediumthrough which light representative of images is directed to a person'seyes. The display may utilize digital light projection, OLEDs, LEDs,uLEDs, liquid crystal on silicon, laser scanning light source, or anycombination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In one embodiment, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface.

FIG. 1A and FIG. 1B depict exemplary system 100 for use in variouscomputer-generated reality technologies, including virtual reality andmixed reality.

In some embodiments, as illustrated in FIG. 1A, system 100 includesdevice 100 a. Device 100 a includes various components, such asprocessor(s) 102, RF circuitry(ies) 104, memory(ies) 106, imagesensor(s) 108, orientation sensor(s) 110, microphone(s) 112, locationsensor(s) 116, speaker(s) 118, display(s) 120, and touch-sensitivesurface(s) 122. These components optionally communicate overcommunication bus(es) 150 of device 100 a.

In some embodiments, elements of system 100 are implemented in a basestation device (e.g., a computing device, such as a remote server,mobile device, or laptop) and other elements of the system 100 areimplemented in a head-mounted display (HMD) device designed to be wornby the user, where the HMD device is in communication with the basestation device. In some examples, device 100 a is implemented in a basestation device or a HMD device.

As illustrated in FIG. 1B, in some embodiments, system 100 includes two(or more) devices in communication, such as through a wired connectionor a wireless connection. First device 100 b (e.g., a base stationdevice) includes processor(s) 102, RF circuitry(ies) 104, andmemory(ies) 106. These components optionally communicate overcommunication bus(es) 150 of device 100 b. Second device 100 c (e.g., ahead-mounted device) includes various components, such as processor(s)102, RF circuitry(ies) 104, memory(ies) 106, image sensor(s) 108,orientation sensor(s) 110, microphone(s) 112, location sensor(s) 116,speaker(s) 118, display(s) 120, and touch-sensitive surface(s) 122.These components optionally communicate over communication bus(es) 150of device 100 c.

In some embodiments, system 100 is a mobile device, such as in theembodiments described with respect to device 100 a in FIGS. 1C-1E. Insome embodiments, system 100 is a head-mounted display (HMD) device,such as in the embodiments described with respect to device 100 a inFIGS. 1F-1H. In some embodiments, system 100 is a wearable HUD device,such as in the embodiments described with respect to device 100 a inFIG. H.

System 100 includes processor(s) 102 and memory(ies) 106. Processor(s)102 include one or more general processors, one or more graphicsprocessors, and/or one or more digital signal processors. In someembodiments, memory(ies) 106 are one or more non-transitorycomputer-readable storage mediums (e.g., flash memory, random accessmemory) that store computer-readable instructions configured to beexecuted by processor(s) 102 to perform the techniques described below.

System 100 includes RF circuitry(ies) 104. RF circuitry(ies) 104optionally include circuitry for communicating with electronic devices,networks, such as the Internet, intranets, and/or a wireless network,such as cellular networks and wireless local area networks (LANs). RFcircuitry(ies) 104 optionally includes circuitry for communicating usingnear-field communication and/or short-range communication, such asBluetooth®.

System 100 includes display(s) 120. In some examples, display(s) 120include a first display (e.g., a left eye display panel) and a seconddisplay (e.g., a right eye display panel), each display for displayingimages to a respective eye of the user. Corresponding images aresimultaneously displayed on the first display and the second display.Optionally, the corresponding images include the same virtual objectsand/or representations of the same physical objects from differentviewpoints, resulting in a parallax effect that provides a user with theillusion of depth of the objects on the displays. In some examples,display(s) 120 include a single display. Corresponding images aresimultaneously displayed on a first area and a second area of the singledisplay for each eye of the user. Optionally, the corresponding imagesinclude the same virtual objects and/or representations of the samephysical objects from different viewpoints, resulting in a parallaxeffect that provides a user with the illusion of depth of the objects onthe single display.

In some embodiments, system 100 includes touch-sensitive surface(s) 122for receiving user inputs, such as tap inputs and swipe inputs. In someexamples, display(s) 120 and touch-sensitive surface(s) 122 formtouch-sensitive display(s).

System 100 includes image sensor(s) 108. Image sensors(s) 108 optionallyinclude one or more visible light image sensor, such as charged coupleddevice (CCD) sensors, and/or complementary metal-oxide-semiconductor(CMOS) sensors operable to obtain images of physical objects from thereal environment. Image sensor(s) also optionally include one or moreinfrared (IR) sensor(s), such as a passive IR sensor or an active IRsensor, for detecting infrared light from the real environment. Forexample, an active IR sensor includes an IR emitter, such as an IR dotemitter, for emitting infrared light into the real environment. Imagesensor(s) 108 also optionally include one or more event camera(s)configured to capture movement of physical objects in the realenvironment. Image sensor(s) 108 also optionally include one or moredepth sensor(s) configured to detect the distance of physical objectsfrom system 100. In some examples, system 100 uses CCD sensors, eventcameras, and depth sensors in combination to detect the physicalenvironment around system 100. In some examples, image sensor(s) 108include a first image sensor and a second image sensor. The first imagesensor and the second image sensor are optionally configured to captureimages of physical objects in the real environment from two distinctperspectives. In some examples, system 100 uses image sensor(s) 108 toreceive user inputs, such as hand gestures. In some examples, system 100uses image sensor(s) 108 to detect the position and orientation ofsystem 100 and/or display(s) 120 in the real environment. For example,system 100 uses image sensor(s) 108 to track the position andorientation of display(s) 120 relative to one or more fixed objects inthe real environment.

In some embodiments, system 100 includes microphones(s) 112. System 100uses microphone(s) 112 to detect sound from the user and/or the realenvironment of the user. In some examples, microphone(s) 112 includes anarray of microphones (including a plurality of microphones) thatoptionally operate in tandem, such as to identify ambient noise or tolocate the source of sound in space of the real environment.

System 100 includes orientation sensor(s) 110 for detecting orientationand/or movement of system 100 and/or display(s) 120. For example, system100 uses orientation sensor(s) 110 to track changes in the positionand/or orientation of system 100 and/or display(s) 120, such as withrespect to physical objects in the real environment. Orientationsensor(s) 110 optionally include one or more gyroscopes and/or one ormore accelerometers.

FIGS. 1C-1E illustrate examples of system 100 in the form of device 100a. In FIGS. 1C-1E, device 100 a is a mobile device, such as a cellularphone. FIG. 1C illustrates device 100 a carrying out a virtual realitytechnique. Device 100 a is displaying, on display 120, a virtualenvironment 160 that includes virtual objects, such as sun 160 a, birds160 b, and beach 160 c. Both the displayed virtual environment 160 andvirtual objects (e.g., 160 a, 160 b, 160 c) of the virtual environment160 are computer-generated imagery. Note that the virtual realityenvironment depicted in FIG. 1C does not include representations ofphysical objects from the real environment 180, such as physical person180 a and physical tree 180 b, even though these elements of realenvironment 180 are within the field of view of image sensor(s) 108 ofdevice 100 a.

FIG. 1D illustrates device 100 a carrying out a mixed reality technique,and in particular an augmented reality technique, using pass-throughvideo. Device 100 a is displaying, on display 120, a representation 170of the real environment 180 with virtual objects. The representation 170of the real environment 180 includes representation 170 a of person 180a and representation 170 b of tree 180 b. For example, the device usesimage sensor(s) 108 to capture images of the real environment 180 thatare passed through for display on display 120. Device 100 a overlays hat160 d, which is a virtual object generated by device 100 a, on the headof the representation 170 a of person 180 a. Device 100 a tracks thelocation and/or orientation of physical objects with respect to theposition and/or orientation of device 100 a to enable virtual objects tointeract with physical objects from the real environment in theaugmented reality environment. In this example, device 100 a accountsfor movements of device 100 a and person 180 a to display hat 160 d asbeing on the head of the representation 170 a of person 180 a, even asdevice 100 a and person 180 a move relative to one another.

FIG. 1E illustrates device 100 a carrying out a mixed reality technique,and in particular an augmented virtuality technique. Device 100 a isdisplaying, on display 120, a virtual environment 160 withrepresentations of physical objects. The virtual environment 160includes virtual objects (e.g., sun 160 a, birds 160 b) andrepresentation 170 a of person 180 a. For example, device 100 a usesimage sensor(s) 108 to capture images of person 180 a in realenvironment 180. Device 100 a places representation 170 a of person 180a in virtual environment 160 for display on display 120. Device 100 aoptionally tracks the location and/or orientation of physical objectswith respect to the position and/or orientation of device 100 a toenable virtual objects to interact with physical objects from realenvironment 180. In this example, device 100 a accounts for movements ofdevice 100 a and person 180 a to display hat 160 d as being on the headof representation 170 a of person 180 a. Notably, in this example,device 100 a does not display a representation of tree 180 b even thoughtree 180 b is also within the field of view of the image sensor(s) ofdevice 100 a, in carrying out the mixed reality technique.

FIGS. 1F-1H illustrate examples of system 100 in the form of device 100a. In FIGS. 1F-1H, device 100 a is a HMD device configured to be worn onthe head of a user, with each eye of the user viewing a respectivedisplay 120 a and 120 b. FIG. 1F illustrates device 100 a carrying out avirtual reality technique. Device 100 a is displaying, on displays 120 aand 120 b, a virtual environment 160 that includes virtual objects, suchas sun 160 a, birds 160 b, and beach 160 c. The displayed virtualenvironment 160 and virtual objects (e.g., 160 a, 160 b, 160 c) arecomputer-generated imagery. In this example, device 100 a simultaneouslydisplays corresponding images on display 120 a and display 120 b. Thecorresponding images include the same virtual environment 160 andvirtual objects (e.g., 160 a, 160 b, 160 c) from different viewpoints,resulting in a parallax effect that provides a user with the illusion ofdepth of the objects on the displays. Note that the virtual realityenvironment depicted in FIG. 1F does not include representations ofphysical objects from the real environment, such as person 180 a andtree 180 b even though person 180 a and tree 180 b are within the fieldof view of the image sensor(s) of device 100 a, in carrying out thevirtual reality technique.

FIG. 1G illustrates device 100 a carrying out an augmented realitytechnique using pass-through video. Device 100 a is displaying, ondisplays 120 a and 120 b, a representation 170 of real environment 180with virtual objects. The representation 170 of real environment 180includes representation 170 a of person 180 a and representation 170 bof tree 180 b. For example, device 100 a uses image sensor(s) 108 tocapture images of the real environment 180 that are passed through fordisplay on displays 120 a and 120 b. Device 100 a is overlaying acomputer-generated hat 160 d (a virtual object) on the head ofrepresentation 170 a of person 180 a for display on each of displays 120a and 120 b. Device 100 a tracks the location and/or orientation ofphysical objects with respect to the position and/or orientation ofdevice 100 a to enable virtual objects to interact with physical objectsfrom real environment 180. In this example, device 100 a accounts formovements of device 100 a and person 180 a to display hat 160 d as beingon the head of representation 170 a of person 180 a.

FIG. 1H illustrates device 100 a carrying out a mixed reality technique,and in particular an augmented virtuality technique, using pass-throughvideo. Device 100 a is displaying, on displays 120 a and 120 b, avirtual environment 160 with representations of physical objects. Thevirtual environment 160 includes virtual objects (e.g., sun 160 a, birds160 b) and representation 170 a of person 180 a. For example, device 100a uses image sensor(s) 108 to capture images of person 180 a. Device 100a places the representation 170 a of the person 180 a in the virtualenvironment for display on displays 120 a and 120 b. Device 100 aoptionally tracks the location and/or orientation of physical objectswith respect to the position and/or orientation of device 100 a toenable virtual objects to interact with physical objects from realenvironment 180. In this example, device 100 a accounts for movements ofdevice 100 a and person 180 a to display hat 160 d as being on the headof the representation 170 a of person 180 a. Notably, in this example,device 100 a does not display a representation of tree 180 b even thoughtree 180 b is also within the field of view of the image sensor(s) 108of device 100 a, in carrying out the mixed reality technique.

FIG. 1I illustrates an example of system 100 in the form of device 100a. In FIG. 1I, device 100 a is a HUD device (e.g., a glasses device)configured to be worn on the head of a user, with each eye of the userviewing a respective heads-up display 120 c and 120 d. FIG. 1Iillustrates device 100 a carrying out an augmented reality techniqueusing heads-up displays 120 c and 120 d. The heads-up displays 120 c and120 d are (at least partially) transparent displays, thus allowing theuser to view the real environment 180 in combination with heads-updisplays 120 c and 120 d. Device 100 a is displaying, on each ofheads-up displays 120 c and 120 d, a virtual hat 160 d (a virtualobject). The device 100 a tracks the location and/or orientation ofphysical objects in the real environment with respect to the positionand/or orientation of device 100 a and with respect to the position ofthe user's eyes to enable virtual objects to interact with physicalobjects from real environment 180. In this example, device 100 aaccounts for movements of device 100 a, movements of the user's eyeswith respect to device 100 a, and movements of person 180 a to displayhat 160 d at locations on displays 120 c and 120 d such that it appearsto the user that the hat 160 d is on the head of person 180 a.

FIG. 2 illustrates an example of physical printed media 200 (e.g., abook). Printed media 200 includes pages 202 a-202 b that depictelectronic products. For example, page 202 a of printed media 200depicts electronic product 204 a (e.g., a mobile phone) and page 202 bof printed media 200 depicts electronic product 204 b (e.g., a desktopcomputer). While shown as a book in FIG. 2, printed media 200 may be anyphysical printed media, such as a magazine, newspaper, flyer, billboard,poster, and so forth.

FIG. 3 illustrates an embodiment of device 100 a displaying, on display120, a representation 300 of printed media 200, meaning that an imagesensor of device 100 a is capturing live image(s) of printed media 200and display 120 is displaying relevant captured image content. Device100 a is an embodiment of system 100, as described in reference to FIGS.1A-1B. In FIG. 3, device 100 a is shown as a mobile device, such as amobile phone. However, it should be understood that device 100 a can beany device configured to display an augmented reality environment, suchas the devices described in reference to FIGS. 1D, 1G, and 1I.

The representation 300 of printed media 200 shown in FIG. 3 includes arepresentation 302 a of page 202 a and a representation 304 a ofelectronic product 204 a. For example, the device 100 a uses one or moreimage sensors (such as image sensor(s) 108 described in reference toFIGS. 1A-1B) to capture images of the printed media 200 that are passedthrough for display on display 120. In some embodiments, when device 100a detects that the captured images include a depiction of electronicproduct 204 a, an affordance is displayed to indicate that the depictionof electronic product 204 a is detected, and that a virtual electronicproduct (e.g., a computer-generated three-dimensional virtual objectcorresponding to the electronic product) is available for display.

In some embodiments, the device 100 a receives a user input 306, such asa tap input, on touch-sensitive surface(s) 122. In some embodiments, theuser input 306 is at a location on the display 120 where therepresentation 304 a of the electronic product 204 a is being displayed.In some embodiments, the user input 306 is at a location of anaffordance indicating the availability of a virtual electronic product(e.g., a computer-generated three-dimensional virtual objectcorresponding to the electronic product) for display. In response toreceiving user input 306, device 100 a overlays or replaces therepresentation 304 a of electronic product 204 a with a virtualelectronic product, as shown in FIGS. 4-6.

In some embodiments (such as with a HMD device or a HUD device), device100 a detects user input 306 by tracking the position of a user's handor finger. When the device 100 a determines that the user's hand orfinger touches the depiction of electronic product 204 a in printedmedia 200, device 100 a overlays or replaces the representation 304 a ofelectronic product 204 a with a virtual electronic product (e.g., acomputer-generated three-dimensional virtual object corresponding to theelectronic product), as shown in FIGS. 4-6.

In some embodiments, device 100 a automatically displays a virtualelectronic product (e.g., a computer-generated three-dimensional virtualobject corresponding to the electronic product) in response to device100 a determining that printed media 200 depicts electronic product 204a. In some embodiments, device 100 a determines printed media 200depicts electronic product 204 a by detecting the appearance ofelectronic product 204 a in images captured by device 100 a (such aswith image sensor(s) 108 described in reference to FIGS. 1A-1B). In someembodiments, device 100 a determines printed media 200 depictselectronic product 204 a by determining that images of the printed media200 include other characteristics associated with electronic product 204a, such as predefined text in the printed media 200, unique symbols(such as a QR code) in the printed media 200, and/or the location andorientation of content (such as text or images) in the printed media200.

FIG. 4 illustrates an embodiment of device 100 a displaying, on display120, a virtual electronic product 404 a overlaying representation 300 ofprinted media 200. Device 100 a is an embodiment of system 100, asdescribed in reference to FIGS. 1A-1B. In FIG. 4, device 100 a is shownas a mobile device, such as a mobile phone. However, it should beunderstood that device 100 a can be any device configured to display anaugmented reality environment, such as the devices described inreference to FIGS. 1D, 1G, and 1I.

As shown in FIG. 4, representation 300 of printed media 200 includesrepresentation 302 a of page 202 a. Virtual electronic product 404 a(e.g., a computer-generated three-dimensional virtual objectcorresponding to the electronic product) overlays the representation 302a of page 202 a. Device 100 a tracks the location and/or orientation ofprinted media 200 with respect to the position and/or orientation ofdevice 100 a to enable the display of virtual electronic product 404 aoverlaying the representation 302 a of page 202 a. In this way, therepresentation 304 a of electronic product 204 a (as shown in FIG. 3)appears to be replaced with virtual electronic product 404 a.

In some embodiments, virtual electronic product 404 a is displayed inresponse to the device 100 a receiving a user input (such as user input306 described in reference to FIG. 3). In some embodiments, virtualelectronic product 404 a is automatically displayed in response todevice 100 a determining that printed media 200 depicts electronicproduct 204 a. In some embodiments, device 100 a determines that printedmedia 200 depicts electronic product 204 a by detecting the appearanceof electronic product 204 a in images captured by device 100 a (such aswith image sensor(s) 108 described in reference to FIGS. 1A-1B). In someembodiments, device 100 a determines printed media 200 depictselectronic product 204 a by determining that images of the printed media200 include other characteristics associated with electronic product 204a, such as predefined text in the printed media 200, unique symbols(such as a QR code) in the printed media 200, and/or the location andorientation of content (such as text or images) in the printed media200.

In some embodiments, before displaying virtual electronic product 404 a,device 100 a determines whether printed media 200 is predefined printedmedia, such as a specific book, a specific page of a book, or a specificdepiction of the electronic product. In some embodiments, device 100 adetermines that printed media 200 is predefined printed media bydetecting that images of the printed media 200 include characteristicsassociated with the predefined printed media, such as predefined text inthe printed media 200, unique symbols (such as a QR code) in the printedmedia 200, and/or the location and orientation of content (such as textor images) in the printed media 200. By determining whether printedmedia 200 is predefined printed media, device 100 a limits access to thevirtual electronic product 404 a to devices having access to thepredefined printed media. Printed media other than the predefinedprinted media that depict the electronic product would not result indevice 100 a displaying virtual electronic product 404 a. Alternativelyor in addition, in some embodiments, when predefined printed media isdetected, device 100 a provides additional information about thepredefined printed media and/or electronic products depicted in thepredefined printed media, in addition to the virtual electronic product404 a (e.g., extra content is provided when device 100 a has access tothe predefined printed media).

In some embodiments, access to virtual electronic product 404 a islimited to predefined depictions of electronic product 204 a in printedmedia 200. For example, device 100 a determines whether the depiction ofelectronic product 204 a in printed media 200 includes a depiction of ascreen of electronic product 204 a before displaying virtual electronicproduct 404 a. If the depiction of electronic product 204 a in printedmedia 200 is from a point of view not showing the screen (e.g., topview, side view, rear view), then the virtual electronic product 404 ais not displayed.

In cases where electronic product 204 a is a depiction of a physicalelectronic product that includes a software feature (e.g., a userinterface), the displayed virtual electronic product 404 a includes anemulation 410 a of the software feature. The emulated software feature410 a recreates the appearance and operation of the physical electronicproduct's software feature. In some embodiments, the emulated softwarefeature 410 a recreates the appearance of the physical electronicproduct's boot-up sequence. In some embodiments, the emulated softwarefeature 410 a is a predefined video depicting the appearance andoperation of the physical electronic product's software feature.

In some embodiments, virtual electronic product 404 a and/or emulatedsoftware feature 410 a are modified in response to device 100 adetecting a user interaction. In some embodiments, device 100 a detectsthe user interaction when a user contacts touch-sensitive surface(s)122. For example, when device 100 a detects a contact on touch-sensitivesurface(s) 122 at a location on display 120 where the virtual electronicproduct 404 a is being displayed, device 100 a interprets the contact asa user interaction with virtual electronic product 404 a and/or emulatedsoftware feature 410 a. In this way, device 100 a treats interactionswith touch-sensitive surface(s) 122 as interactions with virtualelectronic product 404 a and/or emulated software feature 410 a.

In some embodiments (such as with a HMD device or HUD device), device100 a detects the user interaction by tracking the position of a user'shand or finger. For example, when the device 100 a determines that theuser's hand or finger touches the depiction of electronic product 204 ain printed media 200, device 100 a interprets the user's hand or fingerplacement as a user interaction with virtual electronic product 404 aand/or emulated software feature 410 a.

The virtual electronic product 404 a and/or emulated software feature410 a responds to the user interaction in a manner resembling a userinteraction with the physical electronic product. For example, whendevice 100 a detects a user interaction on display 120 at a locationwhere a power button of virtual electronic product 404 a is beingdisplayed, virtual electronic product 404 a is modified to correspond tohow the physical electronic product would respond to a user interactionwith a physical power button (e.g., turn the screen on or off). Whendevice 100 a detects a user interaction on display 120 at a locationwhere an interactive element of emulated software feature 410 a is beingdisplayed (e.g. “continue” element shown in FIG. 4), emulated softwarefeature 410 a is modified to correspond to the selected element (e.g.,the emulated user interface is modified to display historicalinformation about the electronic product, as shown in FIG. 5).

In some embodiments, device 100 a modifies virtual electronic product404 a based on the depiction of electronic product 204 a in printedmedia 200. For example, if the printed media 200 depicts a top view ofthe electronic product 204 a, then the displayed virtual electronicproduct 404 a is rotated to correspond to the orientation being depictedin the printed media 200. In some embodiments, the orientation ofvirtual electronic product 404 a is modified (e.g., rotated) in responseto a user interaction, such as a swipe on touch-sensitive surface(s)122.

FIG. 5 illustrates an embodiment of device 100 a displaying, on display120, virtual electronic product 404 a after a user interaction withemulated software feature 410 a (shown in FIG. 4). Device 100 a is anembodiment of system 100, as described in reference to FIGS. 1A-1B. InFIG. 5, device 100 a is shown as a mobile device, such as a mobilephone. However, it should be understood that device 100 a can be anydevice configured to display an augmented reality environment, such asthe devices described in reference to FIGS. 1D, 1G, and 1I.

As shown in FIG. 5, in response to detecting a user interaction withemulated software feature 410 a (shown in FIG. 4), device 100 a modifiesemulated software feature 410 a and displays emulated software feature410 b. For example, when device 100 a detects a user interaction ondisplay 120 at a location where an interactive element of emulatedsoftware feature 410 a is being displayed (e.g. “continue” element shownin FIG. 4), emulated software feature 410 b corresponding to theselected element is displayed (e.g., the emulated user interface ismodified to display historical information about the electronic product,as shown in FIG. 5). As another example, if device 100 a detects a userinteraction at a location of an application icon being displayed as partemulated software feature 410 a, then device 100 a displays emulatedsoftware feature 410 b corresponding to an emulated version of theapplication being launched within virtual electronic product 404 a(e.g., emulated software feature 410 b corresponds to a software featureof the launched application).

FIG. 6 illustrates another embodiment of device 100 a displaying, ondisplay 120, a virtual electronic product 404 a overlayingrepresentation 300 of printed media 200. Device 100 a is an embodimentof system 100, as described in reference to FIGS. 1A-1B. In FIG. 6,device 100 a is shown as a mobile device, such as a mobile phone.However, it should be understood that device 100 a can be any deviceconfigured to display an augmented reality environment, such as thedevices described in reference to FIGS. 1D, 1G, and 1I.

As shown in FIG. 6, representation 300 of printed media 200 includesrepresentation 302 a of page 202 a. In contrast to FIG. 4, virtualelectronic product 404 a (e.g., a computer-generated three-dimensionalvirtual object corresponding to the electronic product) is displayedsuch that it appears to be hovering above the representation 304 a ofelectronic product 204 a. Device 100 a tracks the location and/ororientation of printed media 200 with respect to the position and/ororientation of device 100 a to enable the display of virtual electronicproduct 404 a hovering above the representation 304 a of electronicproduct 204 a. Device 100 a detects user interactions with virtualelectronic product 404 a as described in reference to FIGS. 4-5.

FIG. 7 illustrates an exemplary technique 700 for providing a softwarefeature of an electronic product (e.g., a computer, mobile phone,tablet, watch, or video game console) in an augmented realityenvironment. In some embodiments, the technique is carried out by system100 described in reference to FIGS. 1A-6. At block 702, images areobtained using one or more image sensors (e.g., image sensor(s) 108described in reference to FIGS. 1A-1B).

At block 704, a determination is made whether the obtained imagesinclude printed media (e.g., printed media 200 described in reference toFIGS. 2-6) depicting the electronic product (e.g., electronic product204 a described in reference to FIGS. 2-6).

At block 706, in accordance with a first set of one or more conditionsbeing satisfied, a virtual object corresponding to the electronicproduct (e.g., virtual electronic product 404 a described in referenceto FIGS. 2-6) is displayed in the AR environment. The first set of oneor more conditions includes a condition that is satisfied when theobtained images include the printed media depicting the electronicproduct. In some embodiments, the first set of one or more conditionsalso includes a condition that is satisfied when the printed mediacorresponds to predefined printed media. In some embodiments, thepredefined printed media is a predefined book, a predefined page of apredefined book, or a predefined depiction of the electronic product. Insome embodiments, the first set of one or more conditions includes acondition that is satisfied when the depiction of the electronic productin the printed media includes a depiction of a screen of the electronicproduct. In some embodiments, the first set of one or more conditionsincludes a condition that is satisfied that is satisfied when a userinput is detected (e.g., user input 306 described in reference to FIG.3).

At block 708, in accordance with a second set of one or more conditionsbeing satisfied, the software feature of the electronic product isprovided with the virtual object. The second set of one or moreconditions includes a condition that is satisfied when the virtualobject is displayed. In some embodiments, the second set of one or moreconditions also includes a condition that is satisfied when a userinteraction with the virtual object is detected (e.g., detecting a userinteraction at a location corresponding to a virtual power button). Insome embodiments, providing the software feature includes displaying,with the virtual object, an artificial boot-up sequence that depicts aboot-up sequence of the physical electronic product. In someembodiments, the software feature is provided by displaying, with thevirtual object, a video depicting one or more aspects of the softwarefeature (e.g., a video that recreates the appearance of the product'sboot-up sequence).

At block 710, a user interaction with the software feature is detected.In some embodiments, the user interaction is an touch contact on atouch-screen display at a location corresponding to the displayedsoftware feature (e.g., a tap gesture is detected on the touch-screendisplay at a location corresponding to where the software feature isdisplayed). In some embodiments, the user interaction is detected bydetecting that a physical position of the user's hand or fingercorresponds to a position in the AR environment of the software feature(e.g., the physical position of the user's hand or finger corresponds towhere the virtual object appears to be positioned, such as on a page ofthe printed media).

At block 712, the virtual object corresponding to the electronic productis modified based on the user interaction. For example, an applicationis launched, additional information is displayed, or a user interface ismodified based on the user interaction. In some embodiments, inaccordance with detecting a first type of user interaction with thevirtual object, the virtual object is modified according to a firstfunction, and in accordance with detecting a second type of userinteraction (different from the first type) with the virtual object, thevirtual object is modified according to a second function that isdifferent from the first function. For example, the virtual object ismodified to provide different feedback based on different locations ofuser touch inputs on the virtual object.

While the present disclosure has been shown and described with referenceto the embodiments provided herein, it will be understood by thoseskilled in the art that various changes in form and details may be madewithout departing from the scope of the present disclosure.

What is claimed is:
 1. A device for providing a software feature of anelectronic product in an augmented reality (AR) environment, comprising:one or more processors; and memory storing one or more programsconfigured to be executed by the one or more processors, the one or moreprograms including instructions for: obtaining, using one or more imagesensors, images depicting a screen of the electronic product; displayinga virtual object corresponding to the electronic product in the ARenvironment, wherein the virtual object includes a virtual screenoverlaying the screen of the electronic product; displaying, with thevirtual screen of the virtual object, the software feature of theelectronic product; detecting a user interaction with the softwarefeature; and modifying the virtual object corresponding to theelectronic product based on the user interaction.
 2. The device of claim1, wherein detecting the user interaction comprises: detecting a touchcontact on a touch-screen display at a location corresponding to theprovided software feature.
 3. The device of claim 1, wherein detectingthe user interaction comprises: detecting that a physical position ofthe user's hand or finger corresponds to a position in the ARenvironment of the software feature.
 4. The device of claim 1, whereinthe images depicting the screen of the electronic product are images ofprinted media.
 5. The device of claim 4, wherein the printed media is apredefined book, a predefined page of a predefined book, or a predefineddepiction of the electronic product.
 6. The device of claim 1, whereindisplaying the software feature of the electronic product includesdisplaying, with the virtual screen of the virtual object, an artificialboot-up sequence that depicts a boot-up sequence of the electronicproduct.
 7. The device of claim 1, wherein displaying the softwarefeature of the electronic product includes displaying, with the virtualscreen of the virtual object, a video depicting one or more aspects ofthe software feature.
 8. A non-transitory computer-readable storagemedium storing one or more programs configured to be executed by one ormore processors, the one or more programs including instructions for:obtaining, using one or more image sensors, images depicting a screen ofthe electronic product; displaying a virtual object corresponding to theelectronic product in the AR environment, wherein the virtual objectincludes a virtual screen overlaying the screen of the electronicproduct; displaying, with the virtual screen of the virtual object, thesoftware feature of the electronic product; detecting a user interactionwith the software feature; and modifying the virtual objectcorresponding to the electronic product based on the user interaction.9. The non-transitory computer-readable storage medium of claim 8,wherein detecting the user interaction comprises: detecting a touchcontact on a touch-screen display at a location corresponding to theprovided software feature.
 10. The non-transitory computer-readablestorage medium of claim 8, wherein detecting the user interactioncomprises: detecting that a physical position of the user's hand orfinger corresponds to a position in the AR environment of the softwarefeature.
 11. The non-transitory computer-readable storage medium ofclaim 8, wherein the images depicting the screen of the electronicproduct are images of printed media.
 12. The non-transitorycomputer-readable storage medium of claim 11, wherein the printed mediais a predefined book, a predefined page of a predefined book, or apredefined depiction of the electronic product.
 13. The non-transitorycomputer-readable storage medium of claim 8, wherein displaying thesoftware feature of the electronic product includes displaying, with thevirtual screen of the virtual object, an artificial boot-up sequencethat depicts a boot-up sequence of the electronic product.
 14. Thenon-transitory computer-readable storage medium of claim 8, whereindisplaying the software feature of the electronic product includesdisplaying, with the virtual screen of the virtual object, a videodepicting one or more aspects of the software feature.
 15. A method forproviding a software feature of an electronic product in an augmentedreality (AR) environment, the method comprising: obtaining, using one ormore image sensors, images depicting a screen of the electronic product;displaying a virtual object corresponding to the electronic product inthe AR environment, wherein the virtual object includes a virtual screenoverlaying the screen of the electronic product; displaying, with thevirtual screen of the virtual object, the software feature of theelectronic product; detecting a user interaction with the softwarefeature; and modifying the virtual object corresponding to theelectronic product based on the user interaction.
 16. The method ofclaim 15, wherein detecting the user interaction comprises: detecting atouch contact on a touch-screen display at a location corresponding tothe provided software feature.
 17. The method of claim 15, whereindetecting the user interaction comprises: detecting that a physicalposition of the user's hand or finger corresponds to a position in theAR environment of the software feature.
 18. The method of claim 15,wherein the images depicting the screen of the electronic product areimages of printed media.
 19. The method of claim 18, wherein the printedmedia is a predefined book, a predefined page of a predefined book, or apredefined depiction of the electronic product.
 20. The method of claim15, wherein displaying the software feature of the electronic productincludes displaying, with the virtual screen of the virtual object, anartificial boot-up sequence that depicts a boot-up sequence of theelectronic product.
 21. The method of claim 15, wherein displaying thesoftware feature of the electronic product includes displaying, with thevirtual screen of the virtual object, a video depicting one or moreaspects of the software feature.